Getting the base image ready to use#

Installing Apptainer#

Getting the Modulus source code#

Although NVIDIA provides the source code but it involves submodules for PySDF which we can’t access. Submodulus are files or directories that is stored on another repopsitory which the user has access to. Alternatively, we can get the source code of Modulus from their Docker container. The syntax to copy files from Docker container to pwd is as follows:

docker cp container_id:FILE_PATH

But first we need to generate a container ID. For that, we can run the Docker image first. The list of all Docker images:

(base) hell@Dell-Precision-T7910:~/Desktop/PhD/PHD/Nvidia Modulus/Modulus v22.07$ docker image ls
REPOSITORY               TAG                       IMAGE ID       CREATED        SIZE
modulus                  22.07                     c3e6e5db96a5   7 weeks ago    16.7GB
nvcr.io/nvidia/pytorch   22.07-py3                 b665b38ccc0e   8 weeks ago    14.8GB
ubuntu                   focal-20220531            20fffa419e3a   2 months ago   72.8MB
nvidia/cuda              11.0.3-base-ubuntu20.04   d134f267bb7a   3 months ago   122MB
nvcr.io/nvidia/pytorch   22.05-py3                 e3470579ea78   3 months ago   14.6GB

Run the Modulus 22.07 image (use Tab completion).

docker run -it modulus:22.07

Locate the source code.

root@1f9c81c1bd54:/examples# ls /
bin   dev  examples  lib    lib64   media  modulus  opt   rapids  run   srv  tmp  var
boot  etc  home      lib32  libx32  mnt    nvidia   proc  root    sbin  sys  usr  workspace
root@1f9c81c1bd54:/examples# ls /modulus/
CONTRIBUTING.md  LICENSE.txt                        README.md              modulus
Dockerfile       Modulus_overview.png               accompanying_licences  poetry.lock
Dockerfile.user  NVIDIA-OptiX-SDK-7.0.0-linux64.sh  external               pyproject.toml
root@1f9c81c1bd54:/examples#

Now note down the container ID using docker container ls.

base) hell@Dell-Precision-T7910:~/Desktop/PhD/PHD/Nvidia Modulus/Modulus v22.07$ docker container ls
CONTAINER ID   IMAGE           COMMAND                  CREATED          STATUS          PORTS                NAMES
1f9c81c1bd54   modulus:22.07   "/opt/nvidia/nvidia_…"   13 seconds ago   Up 12 seconds   6006/tcp, 8888/tcp   recursing_hodgkin

Now navigate to a directory to store the Source code.

cd modulus_source
docker cp 1f9c81c1bd54:/modulus/

The base image#

Now that we have the source code, we can access the Docker recipe Dockerfile. The first two lines in the recipe tell us that the base image is a PyTorch image version 22.05 from NVIDIA NGC which is a pre-built Docker image repository.

ARG PYT_VER=22.05
FROM nvcr.io/nvidia/pytorch:$PYT_VER-py3

Link to the base image: https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch

We need to Pull the Docker image using docker pull nvcr.io/nvidia/pytorch:22.05-py3. Once the pullis complete, we can see the image in docker image ls.

Getting the base image ready#

Now we will convert the Docker image to an Apptainer image as follows:

apptainer build pytorch.img docker://nvcr.io/nvidia/pytorch:22.05-py3

This is take some time (6.6 GB image). We can also use the Docker image directly into the Apptainer recipe but it would take long time to debug the script, as a single build will take at least 30 minutes. So, I thought this approach is much better because our aim is to obtain an error free Apptainer image.

Converting the recipe#

This is an excellent manual for building an Apptainer image. If someone is used to Docker recipe, then here is a conversion guide.

Now let us write the recipe. One can always refer to this manual for more details. ## Header It tells Apptainer about the base operating system that it should use to build the container.

Bootstrap: localimage
From: pytorch22.05.img
Stage: build

Stages are optional.

Sections#

The order of individual sections deosn’t matter.

  • Runtime: when we use the image

  • Build: when we are building the container

Export environment variables#

For the Docker recipe filter all the commands starting with ENV and add in the %environment section.

# export environment variables for runtime not the build
%environment
    # Specify poetry version
    export POETRY_VERSION=1.1.13
    export LD_LIBRARY_PATH="/modulus/external/lib:${LD_LIBRARY_PATH}" \
        NVIDIA_DRIVER_CAPABILITIES=graphics,compute,utility,video \
        _CUDA_COMPAT_TIMEOUT=90
    #echo "Environment variables exported"
    echo "Nvidia Modulus 22.07 Apptainer image"

These environment variables are exported in the runtime not during the build. So, to access an environment variables during the build you need to redefine it in the %post section. Since this section is executed firstly in the runtime. You can put any message here which you want to print at each startup. Here I am printing “Nvidia Modulus 22.07 Apptainer image”.

Copy files to the image#

# Copy required files
%files
    modulus/pyproject.toml ./
    modulus/poetry.lock ./
    modulus/. /modulus/

Build instructions#

We put all the build instruction in this section.

Update the base-image’s OS and install dependencies using the apt package manager. Some application requires you to put interactive command such as yes/no, country name etc. The -y selects yes as the default choice. The DEBIAN_FRONTEND=noninteractive ensures that the default option is selected in fields such as country name, keyboard type etc.#

%post
    # At this point I gave up with their useless $APPTAINER_ENVIRONMENT
    # echo "export POETRY_VERSION=1.1.13" >> $APPTAINER_ENVIRONMENT
    # Setup git lfs, graphviz gl1(vtk dep)
    echo "Updating the OS"
    apt-get update && DEBIAN_FRONTEND=noninteractive\
        apt-get install -y git-lfs graphviz libgl1 && \
        git lfs install
    pip install poetry==1.1.13

Install build esentials#

These tools are needed for building .so library files simialr to .dll in Windows.

echo "Installing OptiX and CMake"
cd /modulus && /modulus/NVIDIA-OptiX-SDK-7.0.0-linux64.sh --skip-license --include-subdir --prefix=/root
cd /root && wget https://github.com/Kitware/CMake/releases/download/v3.18.2/cmake-3.18.2-Linux-x86_64.tar.gz && tar xvfz cmake-3.18.2-Linux-x86_64.tar.gz

Install CUDA wrapper to easily configure GPUs#

echo "Installing CUDA framework for Neural networks"
pip install git+https://github.com/NVlabs/tiny-cuda-nn/@master#subdirectory=bindings/torch

Install the virtual environment#

The dependencies of the Poetry managed Python virtual environment are stored in pyproject.toml. This file contains build system requirements and information, which are used by pip to build the package.

Once you have finished the download poetry.lock stores the version of each Python package in the Poetry managed virtual environment. This prevents you from automatically getting the latest versions of your dependencies.

cd /modulus && poetry config virtualenvs.create false \
&& poetry install --no-interaction

Install PySDF#

This module enables the user to use the Tessellation library.

cd /modulus/external/pysdf && python setup.py install
pip3 install setuptools==42.0.0 # to use easy_install
python3 -m easy_install /modulus/external/eggs/pysdf-0.1-py3.8-linux-x86_64.egg

Cleanup the image#

rm -rf /root/NVIDIA-OptiX-SDK-7.0.0-linux64 /root/cmake-3.18.2-Linux-x86_64 /modulus/external/pysdf  /modulus/.git*
rm -fv /modulus/setup.py /modulus/setup.cfg /modulus/MANIFEST.in

Add metadata#

%labels
    Author Prakhar
    Version v0.0.1
    MyLabel Modulus 22.07 Apptainer image

Add help#

%help
    Nvidia Modulus Apptainer container.

Save the file as buildimage.

Build the image#

The command to build an image is:

apptainer build IMAGE_NAME DEFINITION_FILE

This command requires the superuser permission sudo. Create a new bash file build_command.sh with the following contents.

sudo apptainer build modulus.img buildimage >> output.txt

This will save all the outputs in output.txt in the pwd. Apptainer will still print the basic information in the terminal. Run the bash file to build the Apptainer image. The resulting image will be around 7.4GB.

Testing the image#

Check if the imports are working for torch, modulus and pysdf.sdf.

(base) hell@Dell-Precision-T7910:~/Desktop/apptainer/nvidia_modulus(docker_extracted)$ apptainer shell modulus.img
Nvidia Modulus 22.07 Apptainer image
Apptainer> python
Python 3.8.13 | packaged by conda-forge | (default, Mar 25 2022, 06:04:10)
[GCC 10.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> import modulus
>>> import pysdf.sdf
>>>
Apptainer> exit